Gross and Budget

We have gross and budget information. So let’s add two columns: profit and percentage return on investment for further analysis.

# add profit and return of investment column
movie_metadata <- movie_metadata %>% 
  mutate(profit = gross - budget,
         return_on_investment_perc = (profit/budget)*100)

Colour

Next, we take a look at the influence of colour vs black and white.

# Get colour display types of movies
table(movie_metadata$color)

## 
##                   Black and White            Color 
##                2              124             3680

Since 3.4% of the data is in black and white, we can remove the color column it.

# remove colour column
movie_metadata <- subset(movie_metadata, select = -c(color))

Language

Let’s have a look at the different languages contained within the dataset.

# Look at different languages
table(movie_metadata$language)

## 
##            Aboriginal     Arabic    Aramaic    Bosnian  Cantonese 
##          2          2          1          1          1          7 
##      Czech     Danish       Dari      Dutch    English   Filipino 
##          1          3          2          3       3644          1 
##     French     German     Hebrew      Hindi  Hungarian Indonesian 
##         34         11          2          5          1          2 
##    Italian   Japanese     Kazakh     Korean   Mandarin       Maya 
##          7         10          1          5         14          1 
##  Mongolian       None  Norwegian    Persian Portuguese   Romanian 
##          1          1          4          3          5          1 
##    Russian    Spanish       Thai Vietnamese       Zulu 
##          1         24          3          1          1

Almost 95% movies are in English, which means this variable is nearly constant. Let’s remove it.

Country

Next, we can look at the different types of countries.

# Look at different countries
table(movie_metadata$country)

## 
##    Afghanistan      Argentina          Aruba      Australia        Belgium 
##              1              3              1             40              1 
##         Brazil         Canada          Chile          China       Colombia 
##              5             63              1             13              1 
## Czech Republic        Denmark        Finland         France        Georgia 
##              3              9              1            103              1 
##        Germany         Greece      Hong Kong        Hungary        Iceland 
##             79              1             13              2              1 
##          India      Indonesia           Iran        Ireland         Israel 
##              5              1              4              7              2 
##          Italy          Japan         Mexico    Netherlands       New Line 
##             11             15             10              3              1 
##    New Zealand         Norway  Official site           Peru    Philippines 
##             11              4              1              1              1 
##         Poland        Romania         Russia   South Africa    South Korea 
##              1              2              3              3              8 
##          Spain         Taiwan       Thailand             UK            USA 
##             22              2              4            316           3025 
##   West Germany 
##              1

Around 79% movies are from USA, 8% from UK, 13% from other countries. So we group other countries together to make this categorical variable with less levels: USA, UK, Others.

# Grouping countries
levels(movie_metadata$country) <- c(levels(movie_metadata$country), "Others")
movie_metadata$country[(movie_metadata$country != 'USA')&(movie_metadata$country != 'UK')] <- 'Others' 
movie_metadata$country <- factor(movie_metadata$country)
table(movie_metadata$country)

## 
## Others     UK    USA 
##    465    316   3025

Now that we’ve cleaned up our dataset, we can now continue to explore our data even further! In the next section we will be looking at genres, movie popularity, gross, profit, and many more other aspects pertinent to our data.

Top Genres

genre = movie_metadata['genres']

# Make genre a dataframe
genre = data.frame(table(genre))

# Order genres based on frequency
genre = genre[order(genre$Freq,decreasing=TRUE),]

# Top 20 genres with the most movies
ggplot(genre[1:20,], aes(x=reorder(factor(genre), Freq), y=Freq, alpha=Freq)) + 
  geom_bar(stat = "identity", fill="maroon") + 
  geom_text(aes(label=Freq),hjust=1.2, size=3.5)+
  xlab("Genre") + 
  ylab("Number of Movies") + 
  ggtitle("Top 20 Genres with the most movies") + 
  coord_flip()

From the above a combination of Comedy, Romance, and Drama appears to be, by far, the most frequent produced genres. As you can see, movies have multiple genres that they are associated with. For analysis purposes, we choose to use the first word in the genre column, as this is likely to be the most accurate description of the movie.

Split Genres

Here we first split the genres into multiple columns and merge them together.

head(movie_metadata$genres)

## [1] "Action|Adventure|Fantasy|Sci-Fi" "Action|Adventure|Fantasy"       
## [3] "Action|Adventure|Thriller"       "Action|Thriller"                
## [5] "Action|Adventure|Sci-Fi"         "Action|Adventure|Romance"

Let’s split the genres separated by “|” into 8 different columns.

# Split on "|"
genres_split <- str_split(movie_metadata$genres, pattern="[|]", n=2)

# Create Matrix
genres_matrix <- do.call(rbind, strsplit(movie_metadata$genres, '[|]'))

# Dataframe of genres
genres_df <- as.data.frame(genres_matrix)

genre_df consists of 8 columns, each with different genres. Let’s have a look at the frequency of ALL the genres.

# Collapse all genres into one column
genres_one_col <- gather(genres_df) %>% 
  select(value)

# Top 30
top30 <- genres_one_col %>%
  group_by(value) %>% 
  tally() %>% 
  filter(n >= 30)

# Plot frequency of first column
top30 %>% 
  ggplot(aes(x=reorder(factor(value), n), y=n, alpha=n)) +
  geom_bar(stat="identity", fill="maroon") +
  geom_text(aes(label=n),hjust=1.2, size=3.5) +
  theme(axis.text.x = element_text(angle = 90, hjust = 1)) +
  xlab("Genre") +
  ylab("Frequency") +
  ggtitle("Movie Genre Frequency") +
  coord_flip()

It is evident that the Drama and Comedy genre are still the most popular to be produced. It is also interesting to note that Romance is fairly lower on the list as in the previous graph. This may imply that most movies predominantly co-occur with Comedy or Drama. Romance movies do not co-occur with any of the other genres as frequently as Comedy and Drama do. Additionally, the fact that Comedy and Drama occurs the most does not necessarily mean that they are the most profitable, returning successful ROI’s. We will try and explore this later on in the report.

Previously we assumed that the first genre is the most applicable, therefore, we choose the first column as the genre for the movie and append it to the dataframe.

# Remove old genre column
movie_metadata <- subset(movie_metadata, select = -c(genres))

# Take first column of genres_df and add it to MAIN df
movie_metadata$genre <- genres_df$V1

How does this distribution look like over the years? Lets have a look at the frequency of genres between the period of 1980 and 2016.

# Plotting the movie genres produced
movie_metadata %>%
 group_by(title_year, genre) %>%
 summarise(count = n()) %>%
 ggplot(aes(title_year, as.factor(genre))) +
 geom_tile(aes(fill=count),colour="white") +
 scale_fill_gradient(low="light blue",high = "dark blue") +
 xlab("Year of Movie") +
 ylab("Genre of Movie") +
 ggtitle("Heat Map of Movie Genres Produced Over the Years") +
  theme(panel.background = element_blank())

We can make one or two remarks from this heatmap: Firstly, we can see that Action, Adventure,Comedy, and Drama are genres that are predominantly used as the first term to associate with a movie. Secondly, we can see that Romance, which was previously high in frequency for co-occurring with Comedy and Drama is now very low. This means that there are very few movies that are predominantly Romance, meaning Romance is mostly the second, third, etc term to describe a movie.

Additionally, we can also see that some genres, like Action and Comedy have picked up over the years. This is evident by looking at the darker shades of blue becoming more prominent in the latter years.

Which Genres are Popular?

In our dataset we have Facebook Likes and IMDB scores associated with a movie. This can help give us an indication of how popular each genre is. There are different types of Facebook likes within the dataset, namely director_facebook_likes, actor_3_facebook_likes, actor_1_facebook_likes, cast_total_facebook_likes, actor_2_facebook_likes, and finally movie_facebook_likes. Let’s add them all together for each movie.

# Add column for total facebook likes
movie_metadata$total_facebook_likes <- movie_metadata$director_facebook_likes + 
  movie_metadata$actor_3_facebook_likes + movie_metadata$actor_1_facebook_likes +
  movie_metadata$cast_total_facebook_likes + movie_metadata$actor_2_facebook_likes + 
  movie_metadata$movie_facebook_likes

Now let’s calculate the average IMDB score, average user votes, average Facebook likes and average number of reviewers for each genre.

# creating a data frame containing avg score, avg votes and avg fb likes
score_votes_likes <- movie_metadata %>% 
  group_by(genre) %>%
  summarise(count = n(),
            avg_score = round(mean(imdb_score), 1),
            avg_votes = mean(num_voted_users),
            avg_facebook_likes = mean(total_facebook_likes),
            avg_reviews = mean(num_user_for_reviews)) %>%
  filter(count > 10)

# arranging data frame by average score
arr_score <- arrange(score_votes_likes, desc(avg_facebook_likes))

# Show data frame
as.data.frame(score_votes_likes)

##          genre count avg_score avg_votes avg_facebook_likes avg_reviews
## 1       Action   937       6.3 145049.12          38762.692    462.1942
## 2    Adventure   356       6.5 133549.07          40494.340    355.0365
## 3    Animation    43       6.7  83318.51          30471.674    202.0698
## 4    Biography   200       7.1 100233.72          41000.900    260.4650
## 5       Comedy   997       6.1  64763.60          23896.765    197.9388
## 6        Crime   247       6.9 130843.95          33857.397    362.5789
## 7  Documentary    32       6.8  16159.19           5623.312    126.3750
## 8        Drama   667       6.8  85462.68          31889.940    319.7301
## 9      Fantasy    37       6.3  87311.14          25452.270    374.7027
## 10      Horror   152       5.7  66091.12          21356.158    406.8750
## 11     Mystery    24       6.6 180824.04          35571.625    570.2500

The most liked genres on Facebook is Biography (41000.900), Adventure (40494.340), and Action (38762.692). The most popular genres, based on IMDB scores, are Biography (7.9), Crime (6.9), Documentary (6.8), and Drama (6.8). It may be interesting to note that previously we saw that Comedy and Drama are produced the most frequently overall - yet it seems like the public scores Biography, Crime, Documentary, and Drama movies consistently high. This may also imply that there could be a vast range in scores for Comedy and Drama, justifying a fairly lower score than the rest. Furthermore, Comedy, Action, Drama have a very high number of voters - which may account for the lower overall average.

IMDB ratings VS Movie Count

Let’s have a look at the IMDB rating distribution on the number of movies that are produced. Below we can see that the data is slightly skewed to the left. We have a concentration of data among 6 out of 10 and a long tail to the left. The vast majority of the movies are given a score between 5 and 7.5, with fewer movies scoring higher than that.

# Plotting the IMDB ratings vs movie count
ggplot(movie_metadata, aes(imdb_score)) +
  geom_histogram(bins = 50) +
  geom_vline(xintercept = mean(movie_metadata$imdb_score,na.rm = TRUE),colour = "blue") +
  ylab("Movie Count") +
  xlab("IMDB Rating") +
  scale_x_continuous(limits = c(0, 10)) +
  ggtitle("IMDB Ratings for Movies") +
  ggplot2::annotate("text", label = "Mean IMDB rating",x = 6.2, y = 50, size = 3, colour = "yellow",angle=90)

The lowest scored movie, titled Justin Bieber: Never Say Never, is 1.6 whereas the highest score is 9.3 for The Shawshank Redemption. The mean of the imdb scores is 6.433288.

Popularity over the years

Let’s take a look at when Facebook started. From the graph it is clear that the number of Facebook likes for movies released post 2004 have increased dramatically. Old movies receive fewer likes, which is most likely due to Facebook marketing newer movies.

#Creating the required subset of data 
movies_pop <- movie_metadata %>%
 select(title_year, movie_facebook_likes) %>%
 filter(title_year > 1980) %>%
 group_by(title_year) %>%
 summarise(avg = mean(movie_facebook_likes)) 

#Generating the popularity vs time plot
 ggplot(movies_pop, aes(x = title_year, y = avg)) +
   geom_point() +
   geom_smooth() + 
   geom_vline(xintercept = c(2004),colour = c("blue")) +
   ylab("Average Facebook Likes") +
   xlab("Years Movie Was Produced") +
   ggplot2::annotate("text", label = "Facebook",x = 2003.5, y = 10000, size = 3, colour = "blue",angle=90)

Facebook Likes VS IMDB Score

# Plotting the Facebook Likes VS IMDB Score
movie_metadata %>% 
  group_by(content_rating) %>% 
  summarise(avg_fb_likes = mean(movie_facebook_likes), avg_imdb_score = mean(imdb_score), num=n()) %>% 
  ggplot() +
    geom_point(aes(x=avg_fb_likes, y=avg_imdb_score, color=content_rating, size=num), stat="identity") + 
  scale_y_continuous(limits = c(0, 10)) +
  xlab("Average Facebook Likes") +
  ylab("Average imdb score")

There are three things to look at with this graph: Average IMDB score, average Facebook likes and the number of movies rated per content-rating. We can see that all movies have (on average) very similar IMDB scores, however, they differ higly on the number of Facebook likes. For example, movies rated PG-13 receive much more Facebook likes (on average) than movies rated NC-17. Movies rated R receive (on average) more likes than movies rated NC-17, but still have relatively similar IMDB scores.

IMDB Scores VS Facebook Likes

We can infer a strong correlation between a movie’s Facebook likes and its IMDB Score. This is expected, as a higher individual rating relates to higher viewer satisfaction; and hence it is expected to see an increase in positive online presence. Initially, this graph was constructed to see if there’d be a difference between viewer enjoyment and movie rating. Movie databases are often critisized for the nature of their rating scales, made by critics and placing priority on sentiments and plot, which may not fully coincide with viewer enjoyment. However, as seen below, this is not the case using IMDB’s Scoring.

#Plotting Facebook likes against IMDB score
ggplot(data = movie_metadata, aes(x = imdb_score, y = movie_facebook_likes)) +
  scale_x_continuous(limits = c(0, 10)) +
  geom_smooth()

Vote Counts

The IMDB rating system was first implemented in the 1990’s. Social media platforms like Facebook had started in the mid 2000’s. It is evident that IMDB caused rapid growth in vote counts which was later amplified with the introduction of Facebook.

#Performing operations on Movies Vote Count over the years
 movies_vote1 <- movie_metadata %>%
  select(title_year, num_voted_users) %>%
  group_by(title_year) %>%
  summarise(count = sum(num_voted_users)) 

 ggplot(movies_vote1, aes(x = title_year, y = count/1000)) +
   geom_bar( stat = "identity") +
   geom_vline(xintercept = c(1990,2004),colour = c("orange","blue")) +
   ylab("Vote count (in thousands)") +
   xlab("Years") +
   scale_x_continuous(limits = c(1980, 2014)) +
   ggplot2::annotate("text", label = "Facebook",x = 2003, y = 21000, size = 3, colour = "blue",angle=90) + 
   ggplot2::annotate("text", label = "IMDB",x = 1989, y = 21000, size = 3, colour = "orange",angle=90)

# scatter plot for user votes vs imdb score
scatter_plot(movie_metadata$num_voted_users, movie_metadata$imdb_score, 
     "User Votes",
     "IMDB Score",
     "Scatter plot for User Votes vs IMDB Score",
     "User Votes:",
     "<br>IMDB Score:",
     "<br>Title:",
     alpha = 0.3)

Top 20 directors with highest average IMDB score

Let’s take a look at the directors. We can see that the top IMDB rated directors have very similar scores (8.1 - 8.6). Tony Kaye has the highest rating of 8.6.

# Displaying the avg_imdb score of directors
movie_metadata %>%
  group_by(director_name) %>%
  summarise(avg_imdb = mean(imdb_score)) %>%
  arrange(desc(avg_imdb)) %>%
  top_n(20, avg_imdb) %>%
  formattable(list(avg_imdb = color_bar("orange")), align = 'l')

Top 20 movies based on its Return on Investment

#Top 20 movies based on its Return on Investment
movie_metadata %>% 
  filter(budget >100000) %>%
  mutate(profit = gross - budget,
         return_on_investment_perc = (profit/budget)*100) %>%
  arrange(desc(profit)) %>% 
  top_n(20, profit) %>%
  ggplot(aes(x=budget/1000000, y=return_on_investment_perc)) + 
  geom_point(size = 2) + 
  geom_smooth(size = 1) + 
  geom_text_repel(aes(label = movie_title), size = 3) + 
  xlab("Budget $million") + 
  ylab("Percent Return on Investment") + 
  ggtitle("20 Most Profitable Movies based on its Return on Investment")

Sucessful directors such as George Lucas also have profitable movies. These are the top 20 movies based on its Percentage Return on Investment. ((profit/budget)*100). Since profit earned by a movie does not give a clear picture about its monetary success over the years (1980 to 2016), this analyses, over the absolute value of the Return on Investment(ROI) across its Budget, would provide better results. It is interesting to note that the ROI is high for low budget films and decreases as the budget of the movie increases.

Most Successful Directors Based on Profit

#Top 20 most successful directors
movie_metadata %>%
        group_by(director_name) %>%
  mutate(profit = gross - budget)%>%
select(director_name, budget, gross, profit) %>%
na.omit() %>% 
summarise(films = n(), budget = sum(as.numeric(budget)), gross = sum(as.numeric(gross)), profit = sum(as.numeric(profit))) %>%
mutate(avg_per_film = profit/films) %>%
arrange(desc(avg_per_film)) %>% 
top_n(20, avg_per_film) %>%
ggplot( aes(x = films, y = avg_per_film/1000000)) + 
geom_point(size = 1, color = "maroon") + 
geom_text_repel(aes(label = director_name), size = 3, color = "maroon") + 
xlab("Number of Films") + ylab("Avg Profit $millions") + 
ggtitle("Most Successful Directors")

These are the top 20 most successful directors based on the average profit earned by the movies they directed. There’s an obvious downward trend between average profit and number of films. This could be because as one makes more movies, one could have more hits and misses, therefore the average goes down. It could also be due to film makers having a more diverse range of movies which could include smaller and/or high budget movies" Looking at the most succesful directors, one can determine that the most successful director produced only one movie (Tim Miller) or creating an array of succesful films with large budgets, such as James Cameron.

Top 20 movies based on its Profit

#Top 20 movies based on its Profit
movie_metadata %>% 
  filter(title_year %in% c(2000:2016)) %>%
  mutate(profit = gross - budget,
         return_on_investment_perc = (profit/budget)*100) %>%
  arrange(desc(profit)) %>% 
  top_n(20, profit) %>%
  ggplot(aes(x=budget/1000000, y=profit/1000000)) + 
  geom_point(size = 2) + 
  geom_smooth(size = 1) + 
  geom_text_repel(aes(label = movie_title), size = 3) + 
  xlab("Budget $million") + 
  ylab("Profit $million") + 
  ggtitle("20 Most Profitable Movies")

These are the top 20 movies based on the Profit earned (Gross Earnings - Budget). It can be inferred from this plot that high budget movies tend to earn more profit. The trend is almost linear, with profit increasing with the increase in budget. When assessing the top 20 movies based on profit, Avatar has the highest profit margin, regioning in a similar area to director James cameron.

The top 20 most popular key word

# Plotting the 20 most popular keywords
movie_metadata %>%
  filter(imdb_score >6 ) %>% #Filter to show the graph only for vote_average values greater than 6
  na.omit() %>% 
  ggplot(aes(x = gross/1000000, y = content_rating), height=0) + 
  geom_jitter(alpha = 0.5, col = "darkgreen") + 
  theme(axis.text.x=element_text(angle = 90, hjust = 1))+
  ggtitle("Gross vs Content Rating") +
  xlab("Gross (millions)") +
  ylab("Rating") +
  geom_smooth() +
  coord_flip()

Sentiment results

#Making years integers
sentiments$year <- strtoi(sentiments$year)

As previously mentioned, thre number of movies dramatically increases from 1980 onwards so the calculated sentiments are filtered to reduce sensitivity of the data.

#Sort by year
sentiments_emotions <- sentiments[with(sentiments, order(year)), ] %>% 
  filter(year >= 1980) %>% 
  select(-positive, -negative)

#Heatmap for sentiments
rnames <- sentiments_emotions[,1]
mat_sentiments <- data.matrix(sentiments_emotions[,2:ncol(sentiments_emotions)])
rownames(mat_sentiments) <- rnames
mat_sentiments <- t(mat_sentiments)

df_sentiment <-  as.data.frame(mat_sentiments)
names_emotions <- c("anger", "anticipation", "disgust","fear","joy","sadness","surprise","trust")

sentiments_graph <- cbind(names_emotions, df_sentiment)

#Heatmap
heatmap.2(mat_sentiments, Rowv=NA, Colv=NA, scale="row", col=colorRampPalette(c("white","darkblue")),  margins=c(5,10), trace = "none") 

What is immeadiately visible is the differences in type of emotions in plot keywords has reduced over the years. The movies

Below the trend of positive and negative sentiment is explored.

#Filtering sentiments graph
sentiments_filter <- sentiments %>%
  filter(year >= 1980)

sentiments_filter %>% 
  ggplot(aes(x = year)) +
  geom_line(aes(y = negative, color = 'postive')) +
  geom_line(aes(y = positive, color = 'negative')) + 
  ylab("Sentiment score") +
  xlab("Year") +
  ggtitle("Positive and negative sentiment of keywords across the years")

Generally speaking, the sentiment of common keywords used has been more positive, but has remained relatively constant, apart from the decrease in both positive and negative sentiment from around 1995 to 2010.